Incorporating Data Context to Cost-Effectively Automate End-to-End Data Wrangling

The process of preparing potentially large and complex data sets for further analysis or manual examination is often called wrangling. In classical warehousing environments, the steps in such a are carried out using Extract-Transform-Load platforms, with significant involvement specifying, configuring tuning many them. typical big applications, we need to ensure that all wrangling steps, including web extraction, selection, integration cleaning, benefit from automation wherever possible. Towards this goal, paper we: (i) introduce notion context, which associates portions target schema extensional types commonly available; (ii) define scalable methodology bootstrap an end-to-end based on profiling; (iii) describe how context used inform several within wrangling, specifically, matching, value format transformation, repair, mapping generation selection optimise accuracy, consistency relevance result; (iv) evaluate approach real estate financial data, showing substantial improvements results automated